Distributional Reinforcement Learning via Moment Matching

We consider the problem of learning a set probability distributions from empirical Bellman dynamics in distributional reinforcement (RL), class state-of-the-art methods that estimate distribution, as opposed to only expectation, total return. formulate method learns finite statistics each return distribution via neural networks, RL literature. Existing however constrain learned predefined functional forms which is both restrictive representation and difficult maintaining statistics. Instead, we learn unrestricted statistics, i.e., deterministic (pseudo-)samples, by leveraging technique hypothesis testing known maximum mean discrepancy (MMD), leads simpler objective amenable backpropagation. Our can be interpreted implicitly matching all orders moments between its target. establish sufficient conditions for contraction operator provide finite-sample analysis samples approximation. Experiments on suite Atari games show our outperforms standard baselines sets new record non-distributed agents.